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ABSTRACT 



Smith et al. (1980) analyzed 475 psychotherapy 
studies and concluded that individuals receiving treatment were ' 
better off than 80 percent of the untreated control groups. These 
studies were . criticized on methodological grounds, particularly for 
failing to enable calculation of an index of effect size. To address 
these methodological issues, 20 published studies cited in the Smith 



study, from two treatment domains, 



i.e. 



the effectiveness of 



client-centered therapy (N=17) and transactional analysis (N=3), were 
coded according to type of therapy, allegiance of investigator, 
diagnosis, estimation methods, comparison treatment, subject 
solicitation, treatment center, and source of means. Effect size and 
population effect size were computed according to the methodology in 
the original study. An analysis of the results showed a- lack of 
hompgeneity of effect size estimates.. The success of client-centered 
therapy and transactional analysis appeared to depend on investigator 
bias, location of treatment, diagnosis, and presence of a comparison 
treatment. Specifically, these two treatment methods appeared to be 
most successful when applied to problems that occur in academic 
settings. Caution is recommended in generalizing the results of 
Smith's study , which overly represented academic problems and 
educational counselors compared to those in medical settings. (BL) 
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Reassessment of Psychotherapy 

INTRODUCTION 

The question of the effectiveness of psychotherapy fcas been exam- 
ined in many studies. Several authors have attempted to synthesize 
the results of these studies quantitatively (Smith and Glass, 1977; 
Smith et al . , 1980) through the use of meta-analysis (Glass, 1976). 
Based on the results of 375 studies, Smith and Glass (1977, p. 754) 

cla/imed that "the 'average client receiving psychotherapy was better 

■ / ■ ' : ' 

off than 75% of the untreated controls." Smith et al . (1980) claimed 
tl/at individuals receiving psychotherapy in any form were better off 
an 80ft of the individuals in untreated control groups after analyz- 
ing an additional 100 studies (475 studies total). 

• / 

/ ... 
/ " 

/ These conclusions soon drew sharp criticism. Several authors 

/ 

(for example Eysenck, 1978) ojected to the quality of the studies 
being synthesized, particularly the nature of the control groups. The 
meta-analysis methods used by Smith et al . (1980) have also been cri- 
ticized. Landman and Dawes (1982) have pointed out that Smith and 
Glass (1977) treated 833 measures of effect size from the 375 studies 
as independent data points for purposes of computing overall average 
effect sizes and in order to regress effect sizes on study charac- 
teristics. To correct this problem. Landman and Dawes (1982) computed 
an average effect size for each study from a sample of the original 
studies. They found that the original conclusions were not substan- 
tially changed^ by this procedure. In their discussion, Landman and 
Dawes (1982) pointed out, however, that 26% of the studies that they 
sampled "failed to report one or more sets of results in a form ena- 
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biing calculation of an index of effect size," The authors implied 
that the lettexs M NS" were often substituted in statistical tables in 
place of sufficient information for the calculation of effect sizes. 
As they mention, Greenwald (1975) has noted two related sources of 
bias — the failure of investigators to report nonsignificant results 
and the failure of journals to publish them, " Using simulation 
methods. Lane and Dunlap (1978) have shown that editorial decisions to 
suppress publication of nonsignificant results may lead to large exag- 
gerations of effect siies in, meta-analyses. 

Hedges (1982) has rai sed several additional methodological issues 
that lead us to question the conclusions of Smith et al. (1980). The 
possibility that a variety of non-homogeneous measures, that are not 
linearly equatable (like apples and oranges) have been lumped together 
Concerns us particularly. Hedges (1982) proposes a* test of homogenei- 
ty of effect sizes. Hedges and Olkin (1983, in press) have developed 
methods for partitioning effect sizes into homogeeous groups. And, 
Hedges (1983, in press) has developed a method for testing the homo- 
geneity of effect sizes between 'and within groups (as in the analysis 
of variance ) • 

One further methodological objection to the Smith et al . (1980) 
conclusions must be raised. Cronbach (1982), among others, has point- 
ed out the need to examine the representativeness of a sample with 
regard to the population that one hopes to generalize to in evaluating 
the effects of a treatment. As pointed out by —Hedges (1982), the 
Glass (1976) meta-analysis procedure ignores issues of inference about 



Reassessment of Psychothe rapy ' PAGE 3 

population treatment effects completely. While Smith et al. (1980) 
make vide ranging inferences about the general (population) effects of 
psychotherapy from the results (sample), they have failed to utilize 
sound statistical inference and probability sampling procedures that 
make valid statistical inference possible. Practically speaking, we 
must ask whether or not these 475 studies are representative of all of 
psychotherapy. 

To address the above issues we selected 20 published studies 
cited in Smith et al . (1980) from two treatment domains,* 
client-centered therapy^and transactional analysis. The overlap of 
these studies with those catalogued by the National Library of Medi-. 
cine was examined. We computed an average effect size for each of the 
20 studies and estimated the population effect size by both the 
methods of Glass (1976) and Hedges (1982). The homogeneity of effect 
sizes was tested and rejected.. . Following the overall test we parti- 
tioned the studies into homgeneous groups. Study characteristics 
within the subgroups were examined. The issue of bias due to editori- 
al decisions to publish only significant results was examined in 
Appendix B. Maximum likelihood techniques for truncated data (Cohen, 
1950) were applied to estimate the population effect of psychotherapy 
in a worst possible case, assuming all studies with significant 
results were selected due to their; statistical significance. We 
estimated, at worst, that editorial decisions could have inflated the 
results by as much as 607t>. 
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Studies concerning the effects of client-centered therapy or 
transactional analysis were selected from the Smith et al (1980) 
bibliography for coding. The selection was based on the presence of 
keywords - (such as "client-centered," "nondirective, 99 or 
"transactional") in the titles indicating one of these orientations. 
We also selected studies if the investigator is- known to be of one of 
the above orientations. The Natipnai Library of Medicine \ data base 
(MEDLINE) was searched for the years 1966 through 1977 to establish 
the thoroughness of our coverage .of ttfe literature. Due to cost and 
time limitations we did not include dissertations or unpublished 
papers in our analyses. 

The selected studies were located in university libraries. We 
then coded study characteristics on an abreviated version of the Smith 
et al (1980) coding form. Those characteristics reported as predic- 
tive of psychotherapy effect size were retained on the form. Any 
information needed for effect size computation w<ts also coded. The 
abreviated form, appears as Appendix A. 



A further modification of coding procedures was necessary due to 

the relatively small number of studies in our sample. The categories 

in the coding form of Appendix A were often not used or accounted for 

\;oo few studies. We therefore attempted to combine similar categories 

\ • ■■ 

and to reduce the number of study characteristics coded. 'Control 

Treatment' was reduced to two categories - 'no treatment' or 'any 
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other form of treatment' • The 'solicitation of clients was reduced 
from 5~ categories to two- 1) presentation in response to advertise- 
ment* or autonomous presentation, or solicited by experimentor and 2) 
committed or referred/ Y/e "did not classify the outcome measures for 
the studies, because we wished to generalize our analysis to all stu- 
dies empl oying either cl ient-centereil therapy or transactional 
analysis, and to all outcome measures typically employed with these 
types of studies. In all, eight characteristics w^re coded, with two 
to five categories each. Table 1 shpws this modified coding scheme. 

In most cases the effect sizes are computed -according to the 
methods given in Glass et al. (1981). This usually involves the 
conversion of independent or paired t statistics to effect size esti- 
mates. When more than one ef£oct was reported in a study we cpmpiited 
the average t statistic in order to avoid the problem of 
non- independence. In some instances proportions (i.e. proportion 
"improved" following, therapy) or nonpa rainetr ic statistics were con- 
verted to effect size estimates by referring to a normal probability 
table. 

\ ' ' ... 

The effect sizes were assumed to have the statistical properties 
of a standardized mean difference, g^, where, 



i *i 



In this equation, Y? and are the experimental and control group 
means on the outcome variable, or variable-combination, of a study and 
is the pooled standard deviation, i.e.. 
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•j • . 

sJn c+ s|n e 
sj = — 

■ N c + N E 

Hedges (1980) has noted, however, that the true effect size, 5, 
estimated by is a noncentral t-variatc. The unbiased estimate of 8 

is g^, where 

*> 

g[ = gi x c(m.) 

The value, 'm^' is the sum of experimental and control group sizes, 
' minus two, for study 'i 1 . The correction, f c(m i ) f is approximated to 
a maximum error of 0.007 at m. = 2 (Hedges, 1981) by the equation: 

.3 

c (m . ) = 1 — 

• " 4m i -l 

Unbiased effect sizes, g! f were thus calculated for all' studies from 

the initial effect size estimates. • , 

Hedges (1982) procedures were then used to test the hypothesis 
that all of the studies shared a common effect size. This procedure 
is based on a general one-way ANOVA comparison of effect sizes within 
and between groups , of studies. The FORTRAN program shown in Appendix 
* C was used to perform the effect size ANOVA according to the computa- 

tional equations given in Hedges (1982). The variance, a, of an 
effect size, 5, is estimated for each effect size with the equation, 

& 2( 8 ..) » -J^LJliL. + _ 1 

where 9 ij ' , corresponds to the i' th study in the j # th category. The* 
reliability, p, of the* outcome measure is set tp 1 for all effect 
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sizes. Other critical values calculated in the analysis of variance 
are g , the weighted estimate of the common effect size for all stu- 
dies* and g'--, the weighted estimate of the common effect size for all 
• J 

studies within a given category, 'j'. The final values calculated by 
the ANOVA program in Appendix C are Hp the sum of squared' deviations 
of effect sizes from the weighted common effect size; H^, the within 
groups sum of squared deviations of effect sizes from their weighted 
group effect size; and % Eg, the between groups stun of squares, H^, Ify 
and Hg are distributed as Chi Square statistics with m, p-1, and mrp 
» degrees of freedom respectively, 'M' equals the number of effect 

sizes, 18, and 'p' equals the number of categories into which studies 
are grouped. 

A cluster analysis was also performed to divide the studies into 
homogenous groups, with a ciommon effect size for each group. This 
analysis converts the effect sizes into standard normal deviates and 
compares the difference between two given « normal deviates, 0^ and Uj 
to a critical difference corresponding to a given probability level 
for observing a difference as large or larger. The FORTRAN program 
used for this analysis is shown in Appendix D. The'program implements 
the procedures outlined by Hedges and Olkin (1983, in press), A vari- 
ance-stabilizing transformation, h^ is first calculated for each 
, effect size according to the equation, 

h { = sirih"" 1 (gi/2/2) 

When experimental and control groups have unequal sizes, Z//2 is 
replaced by 

9 
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\ 

x (1 + N E /N C ) 2 )/(N E /N C ) 

The variance stabilized transformations are then transformed to stan- 
dard normal variates with the equation, 

V { - 2 n[h i - h] 

where 9 n 9 is the square mean root of the sample sizes of k-t studies; 
k = 18, and t - the number of studies with 'outlying' sample sizes, 
i.e., greater than 100 in this case. The are assumed to be normal- 
ly distributed around the mean, h, where 

E C 

The value, n^ is the average sample size for study 'i' - (NV + NV)/2. 

Both disjoint and overlapping cluster analysis was performed. In 
disjoint clustering, the difference between adjacent U^'s is compared 
to significant gap sizes between standard normal order statistics for 
a sample of size 18 and a significance level p. Disjoint clustering 
compares the difference between the highest and lowest of a speci- 
fied cluster to critical ^values for Bonferroni multiple comparisons of 
the range of standard normal variates for significance level , p. The 
critical values were obtained from Tables 1 and 2 of Hedges and Olkin 
(1983, in press), and linear interpolation was performed when the 
exact sample sizes were not given. Significance levels wore initially 
set at 0.05, and increased to 0.2. 
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RESULTS 

• \ ' 

\ 

\ ■ 1 

The Smith, et al . .(1980) bibliography yielded sixteen published 
studies employ ing cl ient-centered therapy and three publ ished studies 
employing transactional analysis. One additional study that, deviated 
somewhat from the cl ient-centered protocol but which used 
client-centered outcome measures was also retained for coding* The 
MEDLINE search yielded 299 citations of articles on the above types of 
therapy. Some of these appeared to be papers on therapeutic technique 
or reviews rather than experimental investigations of treatment ^ 
effects. Of the 299 articles located by MEDLINE, only 2 were also 
cited by Smith et al . (1980). Nine had authors also cited by Smith, 
et al., but different titles. Ten of the articles obtained from the 
Smith et al (1980) bibliography have authors who were not located 
through MEDLINE. 

The number of studies in each category of the coded study charac- 
teristics is shown in Table 1. These numbers are enclosed in 
parentheses. Thus, fifteen of the studies used no treatment as the 
'type of comparison treatment' and three studies gave file comparison 
group some other type of therapy (usually behavioural). The allegi- 
ance of the experiment or was considered ' favorable' to the 
experimental treatment in fourteen of the studies, and 'unfavorable' 
or 'indifferent' in four. The other study characteristics in Table 1 
can be similarily interpreted. In general, a large proportion of stu- 
dies centered on academic or character problems and wore located in 
academic settings. The estimation methods usually were based on 



li 



Reassessment of Psychotherapy 



PAGE 10 



Table 1: Coded characteristics of studiesi categories, category codes, frequencies, 



TYPE OF THERAPY , 

1 = Transactional analysis C3) 

2 = Client-centered (15) 

ALLEGIANCE OF INVES TIGATOR 

1 = Favorable (14) 

2 = Indifferent or 

Unfavorable (4) 



DIAGNOSIS 

1 » Academic difficulties (4) 

2 = Character disorder (5) 

3 = Neurotic (3) 

4 = Psychotic (5) 

5 = Normal (1) 

ESTIMATION METHODS 

1 ■ T-values (9) \ 

2 * Mean-dif ference/S.D. (4) 

3 ■ Chi Square (3) 

4 = Monparametrics (1) 

5 - Other (1) 



\ 



T YPE OF COMPARISON TREATMENT 

1 = No treatment (15) 

2 = Other treatment (3) 

SOLICITATION OF SUBJECTS 

1 = Autonomous or 

Advertisement or 

Solicited by experimentor (12) 

2 = committed or referred (5) 

TYPE OF TREATMENT CENTER 

1 = School (7) 

2 = Detention institution (2) 

3 = Private outpatient (2) 

4 = Inpatient (5) 

5 ■ College mental health (2) 

SOURCE OF MEANS 

1 ■ Unadjusted post-test (8) 

2 ■ Covariance-adj usted (1) 

3 ■ .Residual gains (2) 

4 s^Pre-post differences (4) 

5 = Other C3) 



Table 2: Studies and their characteristics, listed in order of effect size. 



No. Authors and Date Ex 

1 Lazarus, 1966 

2 Ashby. et al, 1957 

3 Alper 8 Kranzler. 1970 

4 Jesness, 1975 

5 Andrews, 1971 

6 Rogers, et al, 1967 

7 Roth, et al, 1967 

8 Carkhuff e Truax. 1965 

9 Truax e Wittmer, 1971 

10 Seeman, et al, 1964 

11 Varblo e Landfield, 1969 

12 Truax, et al, 1966 

13 Truax, 1970 

14 Truax e Tourney, 1971 

15 Coche e Douglas, 1977 

16 Dorfman, 1958 

17 Sheldon e Landsman, 1950 

18 Bocfcstraud, 1973 



Comp. Alleg. Solicit Diag. Locat- ' E..Meth. S. Means. 



1 

1. 

2 

1 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 



1 

2 
1 
1 
1 

2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 



3 
3 
1 
2 
5 
4 
1 
4 
4 
2 
3 
2 
4 
2 
4 
2 
1 
1 



3 
5 
2 
2 



3 
5 
2 
1 
2 
2 
1 
3 
1 
1 
1 
1 
1 
3 
4 
1 
1 
2 
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t-statistics or were directly calculated from the mean differences and 
pooled standard deviations according to the equation given earlier for 
effect sizes. Means were most often bas'ed on unadjusted post-test 
measures. 

Other sharactaristics of the studies were not coded, but are 
noteworthy. Of the 20 articles coded, 95% are /from journals and 5% 
appear as„books. Cases handled by counseling psychologists as opposed 
« to clinical psychologists and psychiatrists^ appear to be over- 
represented. Twenty percent involve juvenile delinquency problems and 
20% academic problems. Forty percent of the studies use subjects 
N under age 18 and an additional 25% use college age persons. . Seventy 
percent used objective tests. 

Table 2 shows the coded characteristics of the studies, individu- 
ally. Studies with negative effect sizes are listed first, while 
studies with high, positive effect sizes are listed last.. With this 
order in mind, it is possible to see that unfavorable experimentor 
bias, a diagnosis of a neurosis, the use of comparison treatments and 
location in a detention institution tend" to be associated with each 
other and to lead to negative^ or lower effect sizes; At the other 
end of. the continuum, the jwo studies with the highest effect sizes 
are unique in that they are both concerned with academic difficulties 
and are located in a regular school setting. For these two studies, 
as well as all other studies with positive effect sizes, the investi- 
gator's bias was favorable to the experimental treatment. 

13 
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Table 3: 



Sample sizes, biased and unbiased effect sizes, the effect size variance, the 
variance-stabilizing transformation, and the standard normal variate transf orma'tion. 



No. 


Authors and Date 


# 


if 
25 


fi 

-.470 


✓ 

H 

-.463 


.082' 


TRANS 


S.N*V. 


1 


Lazarus, 1966 


25 


-.164 


-3.33 


2 


Ashby, et al, 1957 


23 


23 


-.185 


-.182 


.087 


-.064 


-2.35 


3 


Alper s Kranzler, 1970 


9 


9 


-.018 


-.017 


.222 


-.006 


' -1.78 


4 


Jesness, 1975 


427- 


482 


.045 


.045 


.004 


0.016 


-1.56 


5 


Andrews, 1971 a 
Rogers, et al, 1967 


23 


23 :} 


fe .110 


.108 


\ .087 


s0.038 


-1.34 


6 


,24 


24 


.156 


.154 


.084 


0.054 


-1.18 


7 


Rothe, et al, 1967 


52 


52 


.365 


,362 


.03*9 . 


0.128 


-0.45 


8 


Carkhuff s Truax. 1965 , 


70 


74 


.440 


.438 


' .028 


0*155 . 


-0.19 


9 


Truax e Wittmer, 1971 


16 


16 


.464 


.453 


' .128 


0.160*' 


-0.14 


10 


Seeman, et al, 1964 


8 


8 


.515 


.489 


.-258 


0.173 


r-0.01 


11 


Varble 8 Landfield, 1969 


36 


35 


.530 


.524 


• 058 


0.185 


0:ll 


12 


truax', et al, 1966 


30 


30 


.630 


.622 


.070 


0.220 


0.45 


13 


Truax, 1970 


16 


16 


.652 


.636. 


.132 


0,225* 


0.50 " 


14 


Truax e Tourney, 1971 


30 


20 


.771 


.759 


.089 


0.263 


0.88 • 


15 


Coche e Douglas, 1977 - 


25 


29 


.806 


.795 


.080 


0.280 


>1.05 


16 


Dorfman, 1958 


17 


17 


.837 


.818-' 


.128 


0.289 


1.13 


17 


/ Sheldon e Landsman, 1950 


9 


10 


1.265 


1.212 


.253 


0.428 


2.50 


18 


Beckstraud, 1973 „ 


. 33 


38 


1.592 


1.575 


.074- 


0.555 


3.76 



Table 3 shows the principle values in the analysis of variance 

and clustering procedures. Sample sizes ranged from 8 to 482. The 

* 

effect sizes, corrected for bias, ranged from -.463 to 1.575. The 
range was slightly greater without correcting for bias. The weighted 
average of biased effect sizes was .252. The weighted average of 
unbiased effect sizes was .249. The unweighted average of biased 
effect sizes, corresponding to Glas s ' s (1976) method, was .472. The 
weighted average takes the variance of the estimate of each effect 
size into account, which in turn is a function of sample size. The 
largest variance _ was .258 for groups of size 8 and the smallest was 
-.004 for groups ot size 450. The variance stabilized transf orma^ons 
of effect sizes ranged from -.164 to .555 , with a weighted mean of 
.16. The standard normal variates are centered arpjind a mean of 0. 
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Table 4: Analysis of variance for homogeneous effect sizes 



CLASSIFICATION * 
NONE 

TYPE OF THERAPY 

TYPE OF COMPARISON TREATMENT 

ALLEGIANCE OF INVESTIGATOR 

SOLICITATION uF SUBJECTS 

DIAGNOSIS 

TYPE OF TREATMENT CENTER 
ESTIMATION METHODS 
SOURCE OF MEANS 

DISJOINT CLUSTERS 

OVERLAPPING CLUSTERS 
CLUSTER 1 ' ' 
.CLUSTER 2 , 
CLUSTER 3 

COMBINED CLUSTERS 

STUDIES 1-5,6-16,17-18 

i Homogeneous groups 
3 -3 Between group differences 
Nonhomogeneons groups 



NUMBER OF 
CATEGORIES JLj 



2 
2 

2 
5 
5 
5 
5 



61. 5< 



23.34 
32.7 s 
31.4s 

61.5 



P > ,0.8 
2 P < .01, 
4 P < .05, 



57. 2« 
28.3s 
30.45 
49. 7« 
40. 7 « 
33. 6« 
49. 6« 
32. 8« 

33.03 



10.8i 
8.9i 



3 P 
5 P 



.001 
.01, 



4.5 

33.33 

31.23 

11.9a 

20.93 

28.82 

12.02 

28.83 

28.63 



50.93 
52.63 

6 p < .001 or less 



Table 5: Grouping of effect sizes according to cluster analysis. 
Study Numbers: ' 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 

1 . 16. 



17 

18 

18 



Disjoint Clusters 

Cluster 1: 1 

Cluster 2: 

Cluster 3 : 
Overlapping Clusters 

Cluster 1 : 

Cluster 2a 

Cluster 3 : . 1 

Combined Clusters 

Cluster 1: 1 i 

Cluster 2: _i : i£ 

Cluster 3: " ■ 17 18 



2 . : 11 

: 16 



3 
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Table 4 shows the analysis of variance results for various group- 
ings . of studies, based on effect sizes corrected for bias. The 
analysis of variance showed that the effect sizes were not homogeneous 
(H T "a 61,5, p < •00001) None of the classifications used in Table 1 

lead to groupings with homogeneous effect sizes. The lowest within 

J 

groups sum of squares was found for studies grouped according to 
whether the comparison group recieved an alternative type of treat- 
ment. Groupings according to the allegiance of the investigator also 
produced a relatively low sum of squares within groups. Groupings 
according to the solicitation c|f subjects and estimation methods were 
noteably heterogeneous. Significant be tween-group differences were 
found for all classifications except 'type of therapy'. We did not 
estimate aver effect sizes for the categories of Table 1# however, 
because of the xack of effect size homogeneity. 

Table 4 also shows the results of the analysis of variance per- 
formed on the study groupings obtained from the cluster analysis. The 
clusters are "illustrated in Table 5. The disjoint clusters, separated 
by a gap length significant to the 0.2 level, consist of studies 1-16, 
17, and 18. The study numbers are taken from Tables 2 and 3, and 
reflect increasing positive effect sizes. The overlapping clusters, 
formed by critical range sizes at a p of .02, consist of study numbers 
1-16, 2-17 and 5-18. None of the groupings formed by the disjoint 
clustering or the' overlapping .clustering alone were homogeneous, 
according to the analysis of variance. The hypothesis of homogeneity 
was rejected at the .05 to .01 level of confidence. However, we used 
the results of both cluster analyses to form three groups of studies - 
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ji 

1_4 § an d 17-18 which proved to have a small^ within groups sum of 

squares of 9.2, p > .8, and a large between -groups sum .of squares of 
54.1, p < .0001. The hypothesis of homogeneity was not rejected. For 
better interpretabil ity, in view of the association of characteristics 
with effect size magnitude brought out by Table 2, the fifth study was 
added to the lower group giving studies 1-5, 6-16 and 17-18 in the 
three groups. Table 4 shows that this new arrangement further 
decreased N the sum of squares within groups and increased the sum of 
squares between groups. 

* The average effect sizes for the three groups were 0.013 for stu- 
dies 1-5; 0.52 for studies 6-16; and 1.52 for studies 1.52. 

The major characteristics of studiesl-5 were investigator bias 
unfavorable to the experimental treatment and/or the use of alterna- 
tive treatments for the comparison group. Studies in the intemediate 
group seemed to have no unique characteristic. However, the investi- 
gator bias was uniformly favorable to the experimental treatment in 
the intermediate studies. The studies with the highest effect sizes, 
as already mentioned were: concerned with minor academic problems in a 
normal school setting. . 

DISCUSSION 



One of the most striking features in our results is the lack of 
homogeneity of effect size estimates. The hypothesis of homogeneity 
of effect sizes for our 18 studies is rejected at p < .00001. Since 

17 
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our effect sizes represent a subset of the even more wide ranging sam- 
ple cf studies used in the meta-analysis of Smith et al . (1980), we 
expect that if their, data were reanalyzed using our techniques the 
lack of homogeneity would be even more apparent. 

Given that the studies are nan-fibmogeneous we asked if the stu- 
dies could be partitioned into interpretabl e subgroups? Through the 
application of the clustering methods of Hedges and « Olkin (1983, in 
press) we obtained 3 homogeneous groups. For better interpretabil ity 
we moved the study with the smallest effect size in the intermediate 

cluster to the cluster of studies with the smallest effect sizes. The 

i 

resulting three groups are even more homogeneous and can be character- 
ized according to investigator bias, location of treatment, diagnosis, 
and presence or absence of a comparison treatment. The success of 
! client-centered therapy and transactional analysis therefore appears 

to depend substantially on these, and possibly other factors. 

Specifically, client centered therapy and transactional analsysis 
appear to be most successful in the treatment of problems that occur 
in academic settings. Of our three homogeneous groups, the one with 
the highest' effect size, 1.52, consisted of studies that used 
volunteer subjects with jnino-r academic problems (a diagnostic category 
omitted"" in Smith et al., 1980) v Psychotherapy in general, however, 
may be -more effective in academic settings. If we divide the Smith et 
al. (1980, p 118) effect sizes into academic (68%) and nonacademic 
(36%) settings, we find an average effect size of .91 in academic and 
.74 in nonacadenic settings. - From the homogeneous grouping at the 
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other extreme of effect size, we find that client centered therapy and 
transactional analysis offers little or no advantage over other fo'rns 
of therapy and tend to be reported to have no effect by investigators 
nonallegiant to these types of therapy. The 5 studies with the lowest 
average effect size of (KQ9, were reported by investigators allegent 
to other forms of therapy or of unknown allegiance, and usually 
involved alternative treatments for the comparison groups. We note 
that Smith et al. (1980, p. 121) report smaller effect sizes for stu- 
dies conducted by experimenters with opposed or unknown allegiance. 

Our findings differ in several respects from those of Smith, et 
*1 (1980). Smith et al (1980) reported an average effect size of .62^ 
from 150 client-centered effect sizes and an average of .67 from jf8 
transactional analysis effect sizes. Weighting these averages by the 
number of effect sizes in each class we obtain.an average effect? si*e 
of ,63 .from the SmitlTet al. (1980) results. We assume this average 
reflects all comparisons, including comparisons with other types of 
treatment. Our estimate by the same methods, based on all 18-effect 
sizes in our review is only .47. Leaving out , comparisons with \other 
forms of treatment, we obtain an average effect size of only .55. VQur 
finding a lower average effect size, using the sane methods for calcu- 
lation, may be due in part to our 18 studies giving the academic 
setting less representation than did Smith, et al, as a whole. * Our 
use of different methods to obtain an average effect size brings up 
another issue. Hedges' (1980) methods take the variance of the effect 
size estimates, and sample sizes, into account. Thus the large study, 
with over 900 subjects, and a near zero effect size receives much^tre 
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it- 



weight than it would by, Glass's method of simply averaging effect 
sizes. Our weighted average of all eighteen effect sizes is ,249, 
using unbiased effect size estimates* We cannot, however, take ,249 
to be a generalizabl e estiamte givan the results of our analysis of 
variance for effect size homogeneity. Nor do we consider Smith's, et 
al . > estimate of to be gener/ilizable. Our smallest estimate is 
obtained by assuming the worst possible case of editorial censorship 
(Appendix B) • The estimate is ,16 or roughly 25% of the effect size 
we were led to exp/jct feem t.ho Sniith et al . (1980) book, 

j 

Final ly turn to issues of sampling and general izabil ity. We 
found very little overlap of these studies with the MEDLINE data base, 
Y/e have already pointed out that Smith et al . (1980) overrepresented^ 
studies involving academic problems. We also note their overrepr esen- 
tation of counseling psychologists and educational counselors as 
opposed to studies more closely tied to medicine (involving clinical 
psychologists and psychiatrists) • We expect smaller effect sizes in 
the medical setting where emotional problems tend to be longer term 
and more difficult to resolve. Out of\1759 effect sizes, reported in 



Smith/ et ai . (1980), 65% involved therapists with a degree in psy- 
chology, 22% an education degree, and 13% a degree in medicine with 
^* specialization in psychiatry. Yet psychologists and psychiatrists are 
equally represented among the total population of mental health pro- 
fessionals (Goldenberg, 1983, p. 6) - therefore psychiatry has been 
grossly underrepresented here. We may also draw upon the comparisons 
between psychotherapy alone versus a control group reported in a 
separate section of Smith et al. (1980) for studies involving drug 
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treatments* We expect these studies to reflect. the more traditional 
medical setting and, in support of our position, we find that Smith et 
al. (1980, p. 165) report an average effect size of .30 for these stu- 
dies. 

In conclusion, while overzealous therapists and the news media 
are quick to generalize the Smith et al . (1980) conclusions to the 
treatment of the most severe emotional problems, and to all types of 
settings, our results and those of Smith et al . (1980) for studies 
involving cases under medical supervision caution against such 
inferential leaps. Such conclusions must await further analyses of 
the data that address inferences about populations explicitely. 



/ 
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Append A\ 

CjgngJL&l in formation : 



STUDY : 

Publication date: 
Publication form: 



1 2 3 • 4 m 

journal book thesis unpublished 



CLIENTS 



Majoi diagnosis: 

neurotic or complex phobic 1 

simple phobic 2 

psychotic 3 

normal 4 

character disorder 5 

delinquent of felon 6 

habituee 7 

mixed 8 

unknown ? A 
emotional/somatic complaint 10 

handicapped 11 

depressive label 12 

Mean age to nearest year: 

Solicitation • of clients: 

autonomous presentation 

presentation in response to advertisement 

solicited by experimentor 

committed 

referred 

DESIGN 

Group assignment of clients: 

1 2. 3 4 5 

random matching pretest convenience other 

equat ion sample nonrandom 

Group assignment of therapists: 

1 2 3 .4 5 

random' matching nonrandom single not 

therapist applicable 

25 



Number of comparisons in this study: 
Number of this comparison: 

Number of outcome measures within this comparison: 



Treatment : 



placebo 
psychodynamic 

client-centered 
Adlerian 

gestalt ( ... 
systematic desensitization 
cognitive/Ellis 

cognitive/other 
transactional, analysis 
behavior modification 
eclectic/dynamic 
eclectic behavioral 
reality therapy 

vocational/ personal development co unci ling 

cognitive behavioral 

implosion 

hypnotherapy 

other 



7 
8 

b 

12 
13 

IS 
H 

18 
19 



Type of comparison: 



1 



control 



2 

placebo 



treatment 



Type of control group: 

1 2.3 
no waiting intact 
treatment list group 



hospital 
ma intenance 



comparison treatment (listed above): 
Allegiance of E to therapy compared: 



1 

yes 



2 

no 



5 

other 



3 

unknown 



Location of treatment: 



School 
hospital 

mental health center 

other clinic 

other outpatient 

private 

other 

unknown 

college mental health 
prisons- 
resident m. h. facility 



1 
2 
3 

4 • 

i 

7 
8 

fo 
11 



Deration of therapy in hours; 

Duration of therapy in weeks: 
• • 

EFFECT SIZE INFORMATION 



ERIC 



Sample size for treatment group: 
Sample size for comparison group; 

a , , 

Total number in comparison: 
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Number of weeks posttherapy measure was take'n: 

Type of outcome measure: 

Reactivity of measure: 

low 1 
high 2 

Calculation of effect size: 

mean diff /control S.D. 1 

MS within , 9 

MS. total minus treatment j 

probit 4 

chi square ' 5 

T table g ~* 

mean and P j 7 

nonparametrics g 

correlations g 

rawdata 10 

estimates \\ 

other , 12 

Source of means 

unadjusted post-test 1 

covarince adjusted 2 

residual gains; 3 

\^pr^-post differences 4 

Significance of treatment effect: 

Treatment group pre-mean: 

Treatment group pre-standard deviation: 

Treatment group post-mean: 

.treatment group post-standard deviation: 

m ^.Qpmparison group pre-meari": 

l 

Comparison pre-standard deviation: 

^omparison post-mean: 

Comparison post -standard deviation: 

\ 

t\ statistic:. 
F 'Statistic : 

mea^square within, residual , or common: 
Treatment group percentage improved: 

Comparison group percentage improved: « 
Effect size: 
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Maximum likelihood estimates of the population effect size are 
easily oBtained through the use 'of large-sample normal approximations. 

Hedges (1982, p. 492) gives the large-sample normal approximation of 

s 

the effect size distribution. A variance stabil izing transformation 
that renders the variance idependent of the mean has been derived by- 
Hedges and Olkin (1983/ in press). We use this normal approximation 
and variance stabilizing transformation earlier in this paper to per- 
form a cluster analysis. The variance stabilizing transformation has 
been modified by Champney (1983, in preparation) for the case of 
unequal treatment and control group sample sizes. This transformation 
is given in the methods section. The variance of this transformation 
is obtained .from the delta theorem given in Rao (1973, p. 385) : 

2 '2 
where Q i is\the variance of the transformed variate, t (5 i ) is the 

2 

square of the first derivative of the transformation, and a {6^) is 



the variance of 5^. By substitution we obtain: 

Q i = {l/[2(nf + N?)]} 

Note that do^-s^not depend on 5^ Cohen (1950) has derived the 
likelihood of a/truncated normal variable. The log likelihood, as we 
have adapted it to the present problem is: 



^ {-log[r i (5)]-log[Q./(2n)]-[t(g i )-t(5 i: )] 2 /2Q i } 
i=l 
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where k is the number of observed studies and Pi(5) is the area under 
the normal curve corresponding to observed studies. We use this equa- 
tion to estimate a common population effect size 5 from the 
distribution with truncation determined by the * level (say p<.05) 
used as the editorial criterion. To accomplish this we adopt trial 
values of 6 by a grid search method until we obtain a value, S, that 
maximizes the likelihood. The method, its justification and 
rationale, and simulations to examine its small sample behavior are 

described in detail in Champney (1983, in preparation). 

i 

.- 

To apply the method to psychotherapy effect sizes we assume a 
worst possible case situation. All stidues with statistically signi- 
ficant effect sizes are assumed to arise from a truncated normal 
distribution. Nonsignificant effect sizes are assumed., to arise from 
an untruncatea distribution. By adding the likelihoods for these 
effect siz^s we are able to obtain an overall estimate of d. Several 
« levels are examined to determine what * level results in the smal- 
lest estimate of effect size: We also compute S assuming that all of 
the studies arise from an untruncated distribution (i.e. the magni- 
tude of effects is not used as an editorial criterion). The same 18 
studies the we include in the cluster analysis are included here. 

Our results are given in Figure D-l . With an « ievel of .05, 8 
of the effect sizes are statistically significant and our estimate of 
5 is .167. It is apparent from Figure B-l that 5 reaches its smallest 
possible value for . an * level of .08, resulting in 10 significant 
effect sizes and % of .153. Note that as the * level approaches 1.00, 
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S approaches the untruncated case. Our maximum likelihood estimate of 
5 in the untruncated case is .274. This is close to the weighted 
estimate of , .252 that we obtain using Hedges' (1982, p. 494) formula 
without the sample bias correction. With the correction. Hedges for- 
mula e gives an affect size estimate of .249 as indicated previously in 
our paper. As we elaborate in the main body of the paper, the homo- 
geneity assumption under which the above effect size estimates are 
computed is highly unlikely (p<. 000001). 

While the above results serve to illustrate the consequences to 
meta-analyses of basing editorial decisions on the statistical signi- 
ficance of treatment effects we musr caution against the conclusion 
that the results of psychotherapy can be summarized by a single effect 
size estimate. We anticipate that viewing psychotherapy effect size 
in the population as a random variable itself (a Bayesian view) may be 
more appropriate. A Bayesian representation of the distribution of 
population effect sizes is currently under development (Champney, 
1983, in preparation). By assuming some « level as a publication cri- 
terion we are able to estimate the mean and variance of. the effect 
size distribution and in addition the number of studies that were pre- 
vented from publication. Preliminary results suggest that larger mean 
effect sizes are estimated under these conditions. 
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Table B-l: Maximum Likelihood Estimates of Effect Size Assuming 
a Mixture of Studies Including Studies Selected for 
Publication at Signifiance Level Alpha. 



alpha 



effect size estimate 



# significant 



1.00 
.99 
.90 
.50. 
.20 
.10 
.09 . 
.08 
.07 . 
.05 
.01 
.006 
.003 
.001 
.0001 



274 
273 
266 
231 
,182 
,166 
,162 
.158 
.161 
.167 
.212 
.234 
.251 
.248 
.243 



18 
18 
17 
14 
12 
10 
10 
10 
9 
8 
3 
2 
1 
1 
1 
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c »*«***f«*«*M**«M**M.«»»*«<*M»*l»*M*M*M***»«******* •••••• 

C 

C 

C -,. 
C EFFANOV 

C - 
C A FORTRAN/ 77 PROGRAM 

c - 

C FOR 

"C ' 

C ANALYSIS OF VARIANCE OF EFFECT SIZES 

C c • • 

C COMPILED ON A DEC PDP 11/70 MINI COMPUTER 

c wrrn ascii- standard Fortran/ 77 compiler- 

c 

c . 

C .. ■ 

c matihew schulz 

c mesa program 

c department of education , 

C DNIVERSITY OF CHICAGO 

C SEPTEMBER 15, 1982 

C - _ 

£*************«********* |(********«**«***«»»M************t ****** 

c 
c 

C : HOUSEKEEPING ~ 

C • ' 

c 

'IMPLICIT REAL*3(A-II J ,0-Z) 
REAL*8 NC.NE 

DIMENSION ARAY ( 20 , 2 0 ) , BRAY ( 20 , 2 0) , CRAY ( 20 , 20 ) . DRAY ( ZO , 20 ) , 
C ERAY(20),KRAY(20) 

OPEN ( UNIT=3 , NAME= ' EFFECT . LST ' , TYPE= ' NbY/ ' ) ! FOR PRINTOUT 

OPEN (UNIT=1,TYPE=» OLD* ,NAJE= 'EFFECT.DAT' ) ! CONTAINS E. S... N-CONT, N-EXPER 

OPEN ( UN1T=2,TYPE=' OLD '.NAME-' CATEG.DAT') 1C0NTAINS CATEGORY MEMBERSHIP OF STUDIES 

C , 

c _ DATA ENTRY AND INITIAL CALCULATIONS ■ 

C , 

C . F.EAD IN AVERAGE RELIABILITY OF DEPENDENT VARIABLE MEASURES 
C 

R » 1 ! RELIABILITY OF OUTCOiiE MEASURES 

' C 

C WRITE TABLE HEADINGS. FOR PRINTOUT 
. C ' . ' 

WRITE (3, 1000) 
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• C 

C READ IN DATA AND DO INITIAL CALCULATIONS 
C 

20 READ(1,*,END=10)G,NC,NE 

READ(2,*) NCAT ! CATEGORIES NUMBERED 1,2,...,MAXN 

IF(NCAT.GT.MAXN) MAXN=NCAT I THE NUMBER OF CATEGORIES 
KRAY ( NCAT ) =»KRAY ( NCAT ) +1 I COUNTS NUMBER OF E. S. 's PER CATEGORY 
K=KRAY (NCAT) 

A1=(NC+NE)/(NC*NE«R) !ONE .COMPONANT OF THE E.S. VARIANCE 
A2=G»*2/(2*(NC+NE)) I OTHER COMPONANT OF E.S. VARIANCE 
A=A1+A2 I THE VARIANCE OF AN EFFECT' SIZE 
B=l/A !AS ENTERS THE ANOVA EQUATIONS 
C=G/A !AS ENTERS TOE ANOVA EQUATIONS 
D=G**2/A !AS ENTERS TOE ANOVA EQUATIONS 

C 

C STORE THESE VALUES FOR EAOI E.S. 

C _ ... ~ ' 

ARAX (NCAT, K) =A 
BRAY ( NCAT, K)=B 
CRAY ( NCAT, K)=C 
DRAY ( NCAT, K)=D 

C 

c write important values, at this point, to toe printout 
c 

V/RITE(3,2000)NCAT,NC,NE,G,A,B,C,D 

C 

C AND REPEAT FOR EACH E.S. 
C 

GO TO 20 

C 

- C— WEIGHTED E.S. OF CATEGORY 

C 

C ' 
C WRITE HEADINGS FOR SECOND TABLE OF PRINTOUT 

C 

10 WRITE(3,300O; 

c 

C CALCULATE TOE WEIGHTED E.S. FOR EACH CATEGORY 
C 

DO 1, I=1,MA3N. , 

C 

C INITIALIZE NUMERATOR AND DENOMINATOR TO ZERO 
JC 

CTOT=0 
BTOT=0 

c ! 

C THEN TAKE NUMERATOR AND DENOMINATOR SUMMATIONS WITOIN CATEGORIES 
C 

D0^2,- 7*1, KRAY (I) 
CTOT= GTOT+ CRAY ( I , J ) 
BTOT=BTOT+BRAY ( I , J) 
2 CONTINUE 

C • 

C CALCULATE AND STORE VffilGHTED E.S. OF CATEGORY 'I' 

C , 

ERAY(I)=(CTOT/BTOT) 

33 
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C 

1 CONTINUE 
C 

C 
C 

C BEGIN WITU LOWEST, PROCEED TO HIGHEST CATEGORY 
C 

DO 3, I=1,MAXN 

C 

C INITIALIZE TERMS IN ANOVA EQUATIONS TO ZERO 

C \ ' 

BTOT-0 

_ CTOT=0 ; 

DTOT=0 
F=0 

C 

C CALCULATE SUMS AS ABOVE 
C 

DO 4, J=»1,KRAY(I) , 

BTOT=BTOT+BRAY (I.J) 

CTOT=CTOT+CRAY ( I , J ) 

DTOT^DXOT+DRAY (I.J) 

F=F+ERAY ( I ) * *2/ ARAY ( I . J ) 
4 CONTINUE 
C 

C WRITE THESE INTERMEDIATE VALUES TO PRINTOUT TABLE 2 
C 

WRITE (3 ,4000) I , BTOT, Cl'OT, D'l"OT, ERAY ( I ) , F 

C 

C CALCULATE SUMS ACCROSS CATEGORIES FOR ANOVA EQUATIONS 
C 

HAN= 11AN+ CTOT : . 

HAD=HAD+BTOT 

IIA1=HA1+DT0T !THE SUMMED STANDARDIZED E.S.'s 
UB1=HB1+F !THE SUMMED WEIGHTED, STANDARDIZED E.S.'s 
3 CONTINUE 

HT=HA1-(HAN**2/HAD) !THE TOTAL SUM OF SQUARES 

EB=HBl-(EAN»»2/HAD) !THE BETWEEN GROUPS SUM OF SQUARES 
HW=HT-KB !THE WITHIN GROUPS SUM OF SQUARES 

C 

C WRITE FINAL VALUES TO TABLE 3 OF PRINTOUT 
C 

g WRITE (3 ,'5000 j , 

WRITE(3i60UU) ELAN, HAD, HA1,HB1 .HT.HB, DW 
CALL EXIT 
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C 

C- 



- FORMAT STATEMENTS - 



C 

1000 F011MATUH ,T2, 'CATEGORY' ,T15,'NC'»T24»'NE',T32,'G', 

CT41, 'A' ,T50, ' B' ,T59. 'C» ,T68, 'D' ) 
2000 FORMAT (1H ,T5 . 12 ,T13 ,F4.0 .T22.F4 .0.3X.5 ( IX. F7 .3 ) ) 
4000 FORMAT (1H ,T5 . 12 ,T13 .5 (2X. F7.3 ) ) 

3000 FORMAT ( 1U0 , T5 , ' CAT' ,T15. 'BTOT' ,T25, 'CTOT' ,T34. 'DTOT' . 

CT44.'ERAY(I)'.1X.'F') m 
5000 FORMAT (1H0.T5. 'HAN' ,T15. 'HAD' ,T25. 'HA1' ,T35 . 'HB1 ' .T45 

CT55. 'HB'.T63.'HW') 
6000 F0RMAT(1H .T5.F7.3.T15.F7.3.T25.F7.3 .T35.F7.3.T45.F7. 

CT55.F6.3.T63.F6.3) 
C . 



C 



END 
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(:•••••*•— • 

c 
c 
c 

C CLUSTER 
C 

C A FORTRAN/ 77 PROGRAM 

C 

C FOR 

C . 

C GROUPING EFFECT SIZES 

C 

C COMPILED ON A DEC PDP 11/70 USING AS CI I- STANDARD FORTRAN/77 COMPILER 

C 

C 

C MATTHEW SCHULZ . 

C - MESA PROGRAM 

C DEPARTMENT OF EDUCATION 

C UNIVERSITY OF CHICAGO > 

C APRIL 16, 1983 

C 

C 

£*********************»«****************************«•»**• 

C 

c 

C HOUSEKEEPING ; 

C 
C 

IMPLICIT REAL* 8 (A- II. O-Z) 
REAL* 8 .NAVGTO, NNTOT, NAVG, NN, NC ( 50 ) , NE ( 50 ) 
DIMENSION CR1(50),CR2(50),EF1(20,20),EF2(20,20), 
, * NUM1(20,20),NUM2(20,20).U1(20,20),U2(20.20). 
* U(50).n(50),Nl(5O),N2(5O),GU(50KGt5O) 

OPEN(UNIT=l,NAfE='EFFECT.DAT' ,TYPE='OLD' ) 1C0NTAINS E.S.. N-COMT, N-EXPER 

OPEN ( UNIT=2 , NAME= ' CR1 . DAT' , TYPE= ' OLD ' ) lCONTAINS STANDARD NORMAL ORDER STATISTICS 

OPEN (UNIT=3.NAME='CR2. DAT', TYPE=' OLD') ! CONTAINS BONFERRONI STATISTICS 

C 

C I _ -. PRELIMINARY CALCULATIONS 

C 

C READ IN STANDARD NORMAL AND BONFERRONI STATISTICS 
C 

DO 14 1=1,100 

READ(2,*,END=15) CR1(I) \ » ' 

READ(3,*,END=15) CR2(I) ' 

14 CONTINUE 

C * 

C READ m EFFECT SIZES AND GROUP SIZES FOR EACH STUDY 
C 

15 DO 1 K=l,1000 - 
READ(1,*,END=10) G(K) ,NC(K) ,NE(K) 

C 

C COMPUTE UNBIASED EFFECT SIZE - • ' . ; . 

O . 

GU(K)=G(K)*(l-3/,(4*(NE(K)'+NC(K)-i))) 
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C 

C COMPOTE AVERAGE GROUP SIZE IN A STUDY 
C 

NAVG»(NE(K)+NC(K))/2 

C 

C KEEP RUNNING TOTAL OF SQRT(NAVG) EXCEPT FOR VERY LARGE STUDIES 
C 

IF(NAVG.GT.IOO) GO TO 21 
N-N+l 

NNTOT-NNTOT+ SORT ( NAVG ) 

NAVGTO-NAVGTO+NAVG IFOR WEIGHTED CALCULATIONS 

C 

C COMPUTE VARIANCE STABILIZING TRANSFORMATION 
C 

21 A=NC(K)/NE(K) 

DK2=(2'(1+A)**2)/A 
n(K)=GU(K)/SQRT(UK2) 

C 

C EEEP RUNNING TOTAL OF TRANFORMATIONS 
C 

IF(NAVG.LE.100)llim=H'lUl+U(K)'NAVG IFOR WEIGHTED CALCULATIONS 
C DTOT-HTOT+H ( K ) IFOR UNWEIGHTED CALCULATIONS 

C 

1 CONTINUE 

10 K=K-1 I SETS E TO THE -NUMBER OF CASES 

C 

C 

C COMPUTE AVERAGE STUDY SIZE, LEAVING OUT LARGE STUDIES 
C 

NN=?(NN1X)T/N)**2 

WRITE(5,*) 'AVERAGE STUDY SIZE IS ' ,NN 1PR0GRESS CHECK 
C ' 
C COMPUTE AVERAGE OF V.S. TRANSFORMATIONS 
C 

DO 16 1=1, K 

IF((NE(K)+NC(K))/2.LE.100)GO TO 16 

HTOT=HTOT+H(K)*NN 

NAVGTO=NAVGTO+NN 
16 CONTINUE 

HBAR=HTOT/NAVGTO 
C i+ 
C COMPUTE STANDARDIZED NORMAL DEVIATES 
C 

DO 2 1=1, K 

U ( I ) =2 * SO. RT ( NN ) * ( H ( I ) -HB AR ) 

2 CONTINUE 
C 

C WRITE A TABLE OF THE RESULTS UP TO THIS POINT 
C 

WRITE (5, 2 000) 
DO 3 1=1, K 

WRITE(5,2001)I,G(I),GU(I),H(I),U(1),NE(I),NC(I) 

3 CONTINUE . i 
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C »< 4 DISJOINT CLUSTERING 

C ' ' 

N-l miB ADDRESS OF HIE LOWEST SNV IN CURRENT CLUSTER 
J-0 I COUNTER FOR NUMBER OF CLUSTERS 

C 

C COMPARE DIFFERENCES BETWEEN S.N. VS. FROM BOTTOM OF RANGE, UP 
C 

DO 4 I-N.K-l 
DIFF-ABS(UU+1)-U(I)) 
IF(DIFF.GT.CRKK)) GO TO 20 

4 CONTINUE 
I-K 

C 

C WHEN A.SIG. DIFF IS FOUND, STORE CURRENT CLUSTER IN ARRAYS 

c v. .... 

20 J-J+l I THE CLUSTER NUMBER 

f L-0 I COUNTER AND ADDRESS FOR THE NUTTIER OF VALUES IN THE CLUSTER 
DO 5 I2-N.I 
L-L+l 

EF1(J,L)-GU(I2) !THE UNBIASED EFFECT SIZE 

NUM1(J,L)«I2 !THE STUDY NUMBER 

U1(J,D-U(I2) ITIIE STANDARDIZED NORMAL DEVIATE 

5 CONTINUE 

N1(J)=L ITHE NUMBER OF VALUES IN HIE CLUSTER 
IF(I.EQ.K) GO TO 30 

N-I+l ISO THE NEXT CLUSTER BEGINS WITH NEXT HIGHEST SNV 
GO TO 4 

C 

C . WRITE A TABLE SHOWING DISJOINT CLUSTERS 
C 

30 WRITE (5, 3 000) 
DO 6 1=1, J 

WRITE(5,3001> l.Nl(I) 
WRITE (5, 3 002) 
DO 7 L=1,N1(I) 

WRITE(5,3 003)L,NUK1(I,L),EF1(I,L),U1(I,L) 

7 CONTINUE 

6 CONTINUE 
C 

C 

C OVERLAPPING CLUSTERING 

C 
C 

L=0 ITHE NUMBER OF CLUSTERS 

J2=K-1 ITHE HIGHEST ALLOWABLE ADDRESS AT BOTTOM OF A CLUSTER 

C 

C BEGIN WITH LARGEST AND DECREASE^ COMPARE TO LOWEST AND INCREASE 
C 

DO 8 I=E,2.-1 
DO 9 J«1,J2 
DIFF=ABSIU(I)-U(J)) 
IF(DIFF.LE.CR2(K-J+U) GO TO 40 
9 CONTINUE. 

IF(J.EQ.1)G0 ID 50 

8 CONTINUE 
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C 

C STORE CLUSTER IN ARRAYS 
C 

40 L-LU I THE NUMBER OF THE CLUSTER 

N-J IT1IE ADDRESS OF THE LOWEST VALUE IN T1IE CLUSTER 
J2-J-1 ! RESETS TOE MGHEST ALLOWABLE ADDRESS FOR NEXT CLUSTER 
N2U)-W+1 IT1IE NUMBER OF VALUES IN TOE CLUSTER 
DO 11 M-l.I-J+l 

NUM21L,M)-N IT11E ADDRESS" AND STUDY NUMBER 
EF2(L.M)«GU(N) ITI1E UNBIASED EFFECT SIZE 
U2(L.M)-U(N) I THE SNV 
N-N+l 

11 CONTINUE 
IF(J.EQ.l)GO TO 50 
GO TO 8 

C 

C WRITE A TABLE OF TOE OVERLAPPING CLUSTERS 
C 

50 WRITE(5.4000) 
DO 12 I-l.L 
WK1TE(5.4001) I.N2(I) 
. WRITE(5.40O2) 
DO 13 M-1.N2(I) 

WRITE(5.40O3)M.NUM2(I.M).EF2(I.M),U2(I,M) 
13 CONTINUE 

12 CONTINUE 
CALL EXIT 

C 
C 

c FORMAT STATEMENTS 

C 

2000 FORMAT (1H , ' STUDY' ,T13 , 'BIASED' ,T24, 'UNBIASED' ,T39 
*'n'.T51,'U'. 

•T60/N' ,T67. 'N'/1H , 'NUMBER' , T14. ' E. S. ' ,T26 , 'E. S. ' . 
•T35, 'TRANSFORM' ,T47 , ' STD.N. VAR. ' ,T59, 'EXP' .T65. 'CONT' ) 

2001 FORMAT ( III , T3 . 12 .T14.F5 .3 ,T26 .F5.3 ,T37, F5 .3 ,T50. 
•F5.2.T59.F4.0.T67.F4.0) 

3000 FORMAT (// IB 'DISJOINT CLUSTERING ' / ) 

3001 FORMAT (/ in-7T8T x etUSTER-NUI.IBER- i -7-I2-r' — N— - ' -, 

3002 FORMAKIH ,T16. 'MEMBER' ,T25, ' STUDY' ,T34. 'E. S. ' . 
•T43 ,' STD.N. VAR. ') 

3003 FORMAT ( III .T18.I2.T27.I2.T34.F6.3.T45.F6.3 ) 

4000 FORMAT ( / / 1H , ' OVERLAPPING CLUSTERING ' / ) 

4001 FORMAT ( / 1H , T8 , ' CLUSTER NUMBER ' , 12 , ' N- '.12/) 

4002 FORMAT (III ,T16, 'MEMBER' ,T25, ' STUDY! ,T34, 'E.S. ' , 
•T4 3, 'STD.N. VAR. ') 

4003 FORMATUH ,118 ,12 ,T27 . 12 ,T34, F6.3 ,T45 .F6.3 ) 
END 



